Skip to content

refactor!: Adapt to apify-client v3#719

Draft
vdusek wants to merge 24 commits into
masterfrom
adapt-to-apify-client-v3
Draft

refactor!: Adapt to apify-client v3#719
vdusek wants to merge 24 commits into
masterfrom
adapt-to-apify-client-v3

Conversation

@vdusek
Copy link
Copy Markdown
Contributor

@vdusek vdusek commented Dec 23, 2025

Description

Issues

Testing

  • The existing SDK tests pass with apify-python-client v3.

@vdusek vdusek self-assigned this Dec 23, 2025
@github-actions github-actions Bot added this to the 130th sprint - Tooling team milestone Dec 23, 2025
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Dec 23, 2025
@vdusek vdusek added the adhoc Ad-hoc unplanned task added during the sprint. label Dec 23, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 23, 2025

Codecov Report

❌ Patch coverage is 92.10526% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.46%. Comparing base (e236b21) to head (102881c).

Files with missing lines Patch % Lines
src/apify/_actor.py 69.69% 10 Missing ⚠️
src/apify/_charging.py 88.13% 7 Missing ⚠️
...ify/storage_clients/_apify/_api_client_creation.py 66.66% 1 Missing ⚠️
...rc/apify/storage_clients/_apify/_dataset_client.py 66.66% 1 Missing ⚠️
.../storage_clients/_apify/_key_value_store_client.py 80.00% 1 Missing ⚠️
...age_clients/_apify/_request_queue_single_client.py 94.73% 1 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                  @@
##           drop-python-3.10     #719      +/-   ##
====================================================
- Coverage             86.95%   86.46%   -0.49%     
====================================================
  Files                    48       48              
  Lines                  2943     2919      -24     
====================================================
- Hits                   2559     2524      -35     
- Misses                  384      395      +11     
Flag Coverage Δ
e2e 35.86% <61.27%> (-1.89%) ⬇️
integration 57.82% <80.07%> (-1.23%) ⬇️
unit 74.64% <75.93%> (-1.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from a45f20c to 4270cb9 Compare January 2, 2026 12:45
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from c26dd9f to 7d6dbb1 Compare January 9, 2026 14:27
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch 3 times, most recently from 3e14e50 to 12a29ae Compare January 23, 2026 14:25
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from ba4d354 to e094d6a Compare January 27, 2026 14:09
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from 4528205 to 9c44f3b Compare February 23, 2026 11:00
vdusek added a commit to apify/apify-client-python that referenced this pull request Feb 23, 2026
### Summary

This is a major refactoring that introduces fully typed Pydantic models
throughout the client library. The models are generated from the OpenAPI
specifications. All API responses now return typed objects instead of
raw dictionaries.

This follows up on apify/apify-docs#2182.

### Issues

- Closes: #21
- Closes: #481

### Packages

- Add direct dependency on `Pydantic`.
- Removes the dependency on `apify-shared`.
- Add dev dependency
[datamodel-code-generator](https://koxudaxi.github.io/datamodel-code-generator/)
for model generation.

### Key changes

- Uses
[datamodel-code-generator](https://koxudaxi.github.io/datamodel-code-generator/)
tool configured via `pyproject.toml` to generate Pydantic models based
on the [OpenAPI specs](https://docs.apify.com/api/openapi.json).
- Refactors the whole codebase to adopt the new generated models.
- All resource clients now return typed Pydantic models (`Actor`,
`Task`, `Run`, etc.).
- Adds response wrappers for validating and extracting API response
data.
- Updates list methods to return typed pagination models.
- Documentation examples now use typed attribute access.
- Updates the SDK to use the new typed client.
- See the corresponding PR in `apify/apify-sdk-python` for details -
apify/apify-sdk-python#719.
  - It will be merged later.

### Architecture

- Get rid of 3/4/5 levels of inheritance.
- Get rid of inline imports because of circular dependencies.
- I had to utilize `ClientRegistry` to be able to achieve that (because
of resource clients-siblings imports).

### Breaking changes

- Client methods now return Pydantic models instead of dicts.
- Access patterns change from dict-style (`result['key']`) to
attribute-style (`result.key`).

### Test plan

- Updated test concurrency to 16 workers.
- A lot of new tests were implemented - coverage ~95%.
- Unit tests - do not call production API, only for testing utils or
other functionality using mocks.
  - Integration tests - call production API.
- Thanks to the new tests, I was able to do a lot of fixes in the
OpenAPI specs.

### Next steps

- Explore the generation of resource clients using
[openapi-python-client](https://github.com/openapi-generators/openapi-python-client).
- Fully automate model updates based on changes in
[apify-api/openapi](https://github.com/apify/apify-docs/tree/master/apify-api/openapi).
- This will be released as part of the Apify client v3.0.
@vdusek vdusek changed the title chore: Adapt to apify-client v3 [WIP] refactor!: Adapt to apify-client v3 [WIP] Mar 10, 2026
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch 2 times, most recently from 8057e7c to 05a8aba Compare April 16, 2026 13:07
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from a5fb9c8 to fc2dde8 Compare April 28, 2026 14:34
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from fc2dde8 to e3439b4 Compare May 11, 2026 07:53
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch 2 times, most recently from 7ac5f59 to 2019646 Compare May 22, 2026 15:35
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch 2 times, most recently from 4e6ffc3 to d9c5fab Compare May 26, 2026 09:26
vdusek added 2 commits May 26, 2026 15:44
Python 3.10 reaches end-of-life soon and apify-client v3 (next major)
will require Python >= 3.11. Bumping the minimum now lets the SDK use
3.11+ stdlib idioms (`datetime.UTC`, `typing.Self`, `TimeoutError`
alias) and unblocks the apify-client v3 adoption series.

- `pyproject.toml`: `requires-python = ">=3.11"`, drop the 3.10
  classifier, set `[tool.ty.environment]` to 3.11.
- `.github/workflows/_checks.yaml`: drop 3.10 from lint, type-check,
  unit-test, integration-test, and e2e-test matrices.
- Refresh `uv.lock` (removes 3.10-only backport packages).
- Apply ruff autofixes unlocked by the version bump (UP017, UP035,
  UP041) across `src/` and `tests/`.

BREAKING CHANGE: Python 3.10 is no longer supported.
The previous ruff autofix (UP017) hoisted `from datetime import UTC` to
module scope, but `main()` functions passed to `make_actor` are serialized
and run on the Apify platform without access to enclosing-module imports,
so they raised `NameError: name 'UTC' is not defined` at runtime.
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from 4a98a06 to cb9ed09 Compare May 27, 2026 10:59
@vdusek vdusek changed the base branch from master to drop-python-3.10 May 27, 2026 10:59
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from cb9ed09 to 24a6fa8 Compare May 27, 2026 11:01
@vdusek vdusek marked this pull request as ready for review May 27, 2026 13:26
@vdusek vdusek changed the title refactor!: Adapt to apify-client v3 [WIP] refactor!: Adapt to apify-client v3 May 27, 2026
@vdusek vdusek marked this pull request as draft May 27, 2026 13:45
vdusek and others added 9 commits May 28, 2026 10:34
Mirrors the apify-client v3 upgrading guide structure. Further breaking-change sections for
v4 (apify-client v3 adoption, webhook API rework) will be added on top in their respective
branches.
Bump apify-client to v3 (>=3.0.0,<4.0.0) and re-use its models, literals,
and constants instead of maintaining SDK-side duplicates.

- Replace SDK models with apify-client equivalents (e.g. `Run` instead of
  `ActorRun`) and import pricing models from the client; remove the now
  redundant `src/apify/_models.py`.
- Drop the `apify-shared` dependency and define the few needed constants
  locally in `_consts.py`.
- Adapt `Actor.start`/`call`/`call_task` to the v3 `run_timeout` API and
  tolerate platform pricing-info env var omissions.
- Adapt the Apify storage clients (dataset, key-value store, request queue)
  to the v3 client surface.
- Refresh `uv.lock` (apify-client 3.0.2, drops apify-shared, adds
  dnspython/email-validator).
- Align unit, integration, and e2e tests with the v3 type and model changes.

BREAKING CHANGE: requires apify-client v3; SDK-side model and literal
duplicates are removed in favor of those exported by apify-client.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the multiple webhook-related re-exports (`WebhookCondition`, `WebhookCreate`,
`WebhookRepresentation`) with a single SDK-level `Webhook` class that covers both
ad-hoc (`Actor.start`/`Actor.call`) and persistent (`Actor.add_webhook`) use cases.
The public surface in `apify` is now just `Webhook` and `WebhookEventType`.
Allows passing `idempotency_key` either via `Webhook(...)` or directly to `Actor.add_webhook(...)`,
matching the previous ergonomics where the key (often derived from `actor_run_id`) was supplied
at call time. The kwarg overrides the value on the `Webhook` instance when both are set.
The preferred way to set the idempotency key is now on the `Webhook` instance directly. The kwarg
form is kept for backward compatibility with a `DeprecationWarning` and will be removed in 5.0.0.
The preventing-duplicates doc example is updated to the new style.
`__all__` is only meant for public modules to control star-imports and the documented API surface;
this is an internal module, so listing the symbols there has no effect and just adds noise.
Documents the v3 → v4 migration for webhooks: the unified SDK-level `Webhook` class,
the removal of `WebhookCondition`/`WebhookCreate`/`WebhookRepresentation`/`ActorPermissionLevel`
from the public namespace, and the deprecation of the `idempotency_key` kwarg on
`Actor.add_webhook` (removed in v5.0.0).
`min_delay_between_retries` is now forwarded as a `timedelta` instead of being converted to
`min_delay_between_retries_millis` (no longer a real kwarg). The single `timeout` kwarg is
replaced by the four tiered timeouts (`timeout_short`, `timeout_medium`, `timeout_long`,
`timeout_max`) that match the client v3 init signature. v4 upgrading guide updated.
The webhook section that landed in `upgrading_to_v3.md` in 24a6fa8 described an intermediate API
(`WebhookCreate` + `WebhookCondition`) that never shipped publicly. Move the migration story
entirely into `upgrading_to_v4.md` and frame it against the actually-released v3.x `Webhook`
class: response fields dropped, `ignore_ssl_errors` / `do_not_retry` / `idempotency_key` kwargs
on `Actor.add_webhook` moved onto the `Webhook` model, and `WebhookCondition` removed.
@vdusek vdusek force-pushed the adapt-to-apify-client-v3 branch from 1a1043e to a23a8b5 Compare May 28, 2026 08:57
vdusek added 13 commits May 28, 2026 11:17
…iers internally

The previous commit replaced the SDK's single `timeout` kwarg with four pass-through tier
kwargs (`timeout_short` / `_medium` / `_long` / `_max`), leaking the apify-client v3
implementation detail into the SDK surface and breaking the public signature unnecessarily.

Restore the single `timeout: timedelta | None` parameter. Internally, treat it as the medium
tier and derive the other three tiers from the client's default ratios (short = `timeout / 6`,
long = max = `timeout * 12`). The v4 upgrading guide is reworded to reflect that the public
signature is unchanged and only the semantics are explained.
…`Actor.new_client`

Conceptually `max` is the upper cap for exponential timeout growth across retries, so it
should exceed `long` rather than equal it. Bump the max-tier ratio from `*12` to `*24`
(double `long`).
Drop the conditional kwargs-dict construction and call `ApifyClientAsync(token=..., ...)`
directly with inline ternaries falling back to the client's default constants. Reads
top-to-bottom like a regular function call instead of a builder pattern.
The two-member `ActorExitCodes` enum was always dereferenced via `.value`
at every call site. Replace it with two module-level int constants
(`EXIT_CODE_SUCCESS`, `EXIT_CODE_ERROR_USER_FUNCTION_THREW`) for simpler
ergonomics matching the other constants in `_consts.py`.
The SDK-level `Webhook` never used pydantic features (no validation,
serialization, or alias-based parsing — users always construct it from
snake_case kwargs and the SDK reads attributes directly when projecting
to the client's `WebhookRepresentation`). A plain `@dataclass` is a
cleaner fit and matches the convention used by `ProxyInfo` and the
charging models.
The 'Only used by `Actor.add_webhook()`' notes were inaccurate - fields like
`idempotency_key` are also forwarded to the API in regular flows. Trim the
class docstring to match.
The SDK only consumes `pricing_model` and per-event title/price from Actor
pricing info, but `Configuration.actor_pricing_info` was typed as a union of
four apify-client pydantic models that require `apifyMarginPercentage`,
`createdAt`, and `startedAt`. The platform's `APIFY_ACTOR_PRICING_INFO` env
var doesn't guarantee any of those, so the SDK had to inject fake epoch
defaults via `_normalize_actor_pricing_info` to make validation pass.

Replace the workaround with dedicated, minimal SDK types:

- `ActorPricingInfo` (frozen dataclass): `pricing_model` + `charge_events`
  mapping, with an `is_pay_per_event` convenience property.
- `ChargeableEvent` (frozen dataclass): `title` + `price_usd`.
- `parse_actor_pricing_info` validator parses the env var directly into
  the SDK type; `from_client_pricing_info` projects the apify-client model
  returned by `Run.pricingInfo`.

The previous `_charging.py:ActorPricingInfo` (return type of
`ChargingManager.get_pricing_info()`) collided with the new name and is
merged into the same type. `get_pricing_info()` now returns
`ActorPricingInfo | None`; `max_total_charge_usd` and `per_event_prices`
are no longer fields on it (read them from `Actor.configuration` /
`ChargingManager.get_max_total_charge_usd()` and from
`info.charge_events` respectively).

BREAKING CHANGE: `Configuration.actor_pricing_info` is now
`apify._pricing.ActorPricingInfo | None`. `ChargingManager.get_pricing_info()`
returns `ActorPricingInfo | None` instead of a dataclass with
`max_total_charge_usd`, `is_pay_per_event`, and `per_event_prices` fields.
The pricing dataclasses and helpers only exist to serve the charging
manager. Keeping them in a separate module forced a one-way import
between two tightly coupled files; inlining them keeps related code
together and reduces the diff vs master.
Cover the apify-client v3 dependency bump (Run return type, WebhookEventType
literal), the pricing-info reshape, and the Webhook pydantic→dataclass
switch. Tighten section structure to match the apify-client v3 upgrade
guide's concise style.
The `apify-shared` removal silently dropped `ActorEventTypes`, leaving
users on `from apify_shared.consts import ActorEventTypes` with a broken
import and no migration path. Re-export it from `apify` as a string
`Literal`, mirroring how `WebhookEventType` is exposed in v4.

Also note the change in the v4 upgrading guide so existing v3 code that
used the `StrEnum` form has a clear migration path to `apify.Event`.
Reverts the pricing-info refactor (commits 1ce478e and 8f328f9) back to
the post-`6aebd3b` shape. The rewrite introduced 5 user-visible breaking
changes (Configuration field type, ChargingManager.get_pricing_info return
shape, optional return, ChargeableEvent wrapper, removed dataclass fields)
to swap an existing 25-line BeforeValidator workaround for a 150-line
SDK-side type. Not worth the API churn - keep the workaround instead.

`Configuration.actor_pricing_info` is again the discriminated union of
apify-client pricing models with `_normalize_actor_pricing_info` injecting
fake defaults for the metadata fields the platform env var omits.
`ChargingManager.get_pricing_info()` again returns the original four-field
`ActorPricingInfo` dataclass (non-optional). Docs example and tests revert
to the master-style usage; the "Pricing info reshaped" section is dropped
from the v4 upgrading guide.
`Configuration.actor_pricing_info` is back to a discriminated union of
pydantic pricing models, but the models now live in `apify._pricing`
instead of `apify-client`. The SDK copies keep `apify_margin_percentage`,
`created_at`, `started_at`, and per-event `eventDescription` optional, so
the platform's `APIFY_ACTOR_PRICING_INFO` env var deserializes cleanly
without the `_normalize_actor_pricing_info` workaround that injected fake
epoch defaults.

`ChargingManagerImplementation._fetch_pricing_info` converts the
`Run.pricingInfo` it gets from the API (an apify-client model) into the
SDK-side model on the boundary - one `_from_client_pricing_info` helper.

Net effect: no user-visible API change, no env-var hack, ~85 lines of
SDK-side pydantic duplication. The duplication is static (the pricing
schema rarely changes) and isolated to one module.
The pricing pydantic models added in the previous commit only serve the
charging manager — keeping them in a separate `_pricing` module just
forced a one-way import between two tightly coupled files. Inline them
into `_charging.py` so related code lives together.
Base automatically changed from drop-python-3.10 to master May 28, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace custom SDK models with generated models from apify-client Adapt to typed apify-client-python

2 participants